HPC Update 2/22/2024

Next Steps

• Look into storage of files under /work directory instead of /home directory (this should help avoid disk quota errors).
• Make smaller runs, perhaps with Example 6, until we get valid runs working end-to-end. Then we can try bigger (Banklick) runs.

Chama HPC Run All Script

The code for the "run all" script can be found below. To run this script, the following command is used: sbatch hpc_run_all_chama.sh. All Slurm configuration values are defined inside the script itself with #SBATCH tags.

#!/bin/bash

#SBATCH --job-name=chama23
#SBATCH --partition=compute
#SBATCH --mem-per-cpu=32G
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --time=168:00:00

cd /home/jspecht/chama22/chama_django
pip install -r requirements.txt
python manage.py makemigrations
python manage.py migrate
python manage.py run_all

Chama HPC Config YAML

The YAML config file on the HPC remained mostly unchanged. There are two main configuration parameters that were changed when testing:

• num_threads
• end_step

num_threads controls how many threads inside Python are used to generate signal files. Signal file generation is the most resource intensive part of the process and so benefits greatly from such multithreading. While testing, we experimented with different combinations of threads (inside Python) and CPUs (allocated in HESC).

end_step defines the final step in our simulation(s) and so practically controls how large the runs will be. The default for this value is 10,000. While testing, we experimented with different end step values to gauge roughly how long variously sized simulations would take to complete.

Issues Faced

Jake started six total ensembles (Banklick network) on HESC, all of which failed to finish.

1. slurm-70069: The first run failed due to time limit. The default time limit is one hour unless otherwise specified.
2. slurm-70947: The second run was cancelled partway through once Jake realized it would end after an hour.
3. slurm-71041: The third run (and all following runs) was started with a 7 day time limit and failed after a couple hours due to a bug in the code.
4. slurm-71667: The fourth run failed relatively quickly (less than an hour) due to the disk quota being exceeded. After this run, Jake deleted all previous results to clear up disk space.
5. slurm-71670: The fifth run failed after 4:36 hours due to a generic "disk I/O error", which was likely spawned from yet another disk quota error.
6. slurm-71708: The sixth run failed after 3:#9 hours due to a previously unseen Pandas DataFrame error.

Disk Quota Exceeded

Error Stacktraces

1. slurm-70069

*** JOB 70069 ON r2n26 CANCELLED AT 2024-02-16T17:14:22 DUE TO TIME LIMIT ***

2. slurm-70947

This run was cancelled early, so there are no errors to show.

3. slurm-71041

Traceback (most recent call last):
  File "/home/jspecht/chama22/chama_django/manage.py", line 22, in <module>
    main()
  File "/home/jspecht/chama22/chama_django/manage.py", line 18, in main
    execute_from_command_line(sys.argv)
  File "/home/jspecht/.local/lib/python3.9/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/home/jspecht/.local/lib/python3.9/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/jspecht/.local/lib/python3.9/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/jspecht/.local/lib/python3.9/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
  File "/home/jspecht/chama22/chama_django/chama_django/management/commands/run_all.py", line 272, in handle
    optimizer.optimize(combined_signal_file_df=combined_df,
  File "/home/jspecht/chama22/chama_django/chama_django/chama_optimizer.py", line 301, in optimize
    df_ea = pd.DataFrame(entity_assessment_data.items(), columns=['entity', 'sensors'])
NameError: name 'pd' is not defined

4. slurm-71667

Traceback (most recent call last):
  File "/home/jspecht/chama22/chama_django/manage.py", line 22, in <module>
    main()
  File "/home/jspecht/chama22/chama_django/manage.py", line 18, in main
    execute_from_command_line(sys.argv)
  File "/home/jspecht/.local/lib/python3.9/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/home/jspecht/.local/lib/python3.9/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/jspecht/.local/lib/python3.9/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/jspecht/.local/lib/python3.9/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
  File "/home/jspecht/chama22/chama_django/chama_django/management/commands/run_all.py", line 226, in handle
    inp_file_creator.create_input_files(config, ensemble_id)
  File "/home/jspecht/chama22/chama_django/chama_django/chama_inp_generator.py", line 378, in create_input_files
    shutil.copy(original_inp_file, new_inp_file)
  File "/usr/local/apps/oneapi/intelpython/python3.9/lib/python3.9/shutil.py", line 427, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/local/apps/oneapi/intelpython/python3.9/lib/python3.9/shutil.py", line 277, in copyfile
    _fastcopy_sendfile(fsrc, fdst)
  File "/usr/local/apps/oneapi/intelpython/python3.9/lib/python3.9/shutil.py", line 172, in _fastcopy_sendfile
    raise err
  File "/usr/local/apps/oneapi/intelpython/python3.9/lib/python3.9/shutil.py", line 152, in _fastcopy_sendfile
    sent = os.sendfile(outfd, infd, offset, blocksize)
OSError: [Errno 122] Disk quota exceeded: 'network_inputs/BanklickCreekDownstream_base.inp' -> 'ensemble_runs/4/inp_files_out/banklick_S019_2005Jan02T22_wind_direction_60.inp'

5. slurm-71670

Generate Chama inputs 2024-02-21 18:16:52.398005
               Node  ...  banklick_S319_2005Jan02T22_wind_direction_60
0       019.01.0706  ...                                            44
1       019.01.0726  ...                                            44
2       019.01.0732  ...                                            40
3       019.01.0768  ...                                          1073
4       019.01.0770  ...                                          1073
...             ...  ...                                           ...
346545   020.02.030  ...                                             0
346546   020.02.030  ...                                             0
346547   020.02.030  ...                                             0
346548   020.02.030  ...                                             0
346549   020.02.030  ...                                             0

[346550 rows x 353 columns]
Combine signals files 2024-02-21 18:23:56.703093
Traceback (most recent call last):
  File "/home/jspecht/.local/lib/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/home/jspecht/.local/lib/python3.9/site-packages/django/db/backends/sqlite3/base.py", line 328, in execute
    return super().execute(query, params)
sqlite3.OperationalError: disk I/O error

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/jspecht/chama22/chama_django/manage.py", line 22, in <module>
    main()
  File "/home/jspecht/chama22/chama_django/manage.py", line 18, in main
    execute_from_command_line(sys.argv)
  File "/home/jspecht/.local/lib/python3.9/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/home/jspecht/.local/lib/python3.9/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/jspecht/.local/lib/python3.9/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/jspecht/.local/lib/python3.9/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
  File "/home/jspecht/chama22/chama_django/chama_django/management/commands/run_all.py", line 272, in handle
    optimizer.optimize(combined_signal_file_df=combined_df,
  File "/home/jspecht/chama22/chama_django/chama_django/chama_optimizer.py", line 96, in optimize
    ensemble.save()
  File "/home/jspecht/.local/lib/python3.9/site-packages/django/db/models/base.py", line 814, in save
    self.save_base(
  File "/home/jspecht/.local/lib/python3.9/site-packages/django/db/models/base.py", line 877, in save_base
    updated = self._save_table(
  File "/home/jspecht/.local/lib/python3.9/site-packages/django/db/models/base.py", line 990, in _save_table
    updated = self._do_update(
  File "/home/jspecht/.local/lib/python3.9/site-packages/django/db/models/base.py", line 1054, in _do_update
    return filtered._update(values) > 0
  File "/home/jspecht/.local/lib/python3.9/site-packages/django/db/models/query.py", line 1231, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/home/jspecht/.local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1984, in execute_sql
    cursor = super().execute_sql(result_type)
  File "/home/jspecht/.local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1562, in execute_sql
    cursor.execute(sql, params)
  File "/home/jspecht/.local/lib/python3.9/site-packages/django/db/backends/utils.py", line 102, in execute
    return super().execute(sql, params)
  File "/home/jspecht/.local/lib/python3.9/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
  File "/home/jspecht/.local/lib/python3.9/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/home/jspecht/.local/lib/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/home/jspecht/.local/lib/python3.9/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/jspecht/.local/lib/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/home/jspecht/.local/lib/python3.9/site-packages/django/db/backends/sqlite3/base.py", line 328, in execute
    return super().execute(query, params)
django.db.utils.OperationalError: disk I/O error

6. slurm-71708

Traceback (most recent call last):
  File "/home/jspecht/chama22/chama_django/manage.py", line 22, in <module>
    main()
  File "/home/jspecht/chama22/chama_django/manage.py", line 18, in main
    execute_from_command_line(sys.argv)
  File "/home/jspecht/.local/lib/python3.9/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/home/jspecht/.local/lib/python3.9/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/jspecht/.local/lib/python3.9/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/jspecht/.local/lib/python3.9/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
  File "/home/jspecht/chama22/chama_django/chama_django/management/commands/run_all.py", line 258, in handle
    combined_df = combiner.combine_to_df(config, ensemble_id)
  File "/home/jspecht/chama22/chama_django/chama_django/chama_signals_combiner.py", line 45, in combine_to_df
    df_temp = pd.read_csv(csv_file, comment='#')
  File "/usr/local/apps/oneapi/intelpython/python3.9/lib/python3.9/site-packages/pandas/util/_decorators.py", line 211, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/apps/oneapi/intelpython/python3.9/lib/python3.9/site-packages/pandas/util/_decorators.py", line 331, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/apps/oneapi/intelpython/python3.9/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 950, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/usr/local/apps/oneapi/intelpython/python3.9/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 605, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/usr/local/apps/oneapi/intelpython/python3.9/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 1442, in __init__
    self._engine = self._make_engine(f, self.engine)
  File "/usr/local/apps/oneapi/intelpython/python3.9/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 1753, in _make_engine
    return mapping[engine](f, **self.options)
  File "/usr/local/apps/oneapi/intelpython/python3.9/lib/python3.9/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 79, in __init__
    self._reader = parsers.TextReader(src, **kwds)
  File "pandas/_libs/parsers.pyx", line 554, in pandas._libs.parsers.TextReader.__cinit__
pandas.errors.EmptyDataError: No columns to parse from file